Detection of user behavior deviation from defined user groups

ABSTRACT

A machine learning-based technique for user behavior analysis that detects when users deviate from expected behavior. In this approach, a set of user groups are provided, preferably based on information provided from a user registry. A set of training data for each of the set of user groups is then obtained, preferably by collecting security events generated for a collection of the users over a given time period (e.g., a last thirty (30) days). A machine learning system is then trained using the set of training data to produce a model that includes a set of clusters in user behavior model, wherein a cluster is a learned user group that corresponds to a defined user group. Once the model is built, it is used to identify users that deviate from their expected group behavior. In particular, the system compares a current behavior of a user against the model and flags anomalous behavior. The user behavior analysis may be implemented in a security platform, such as a SIEM.

BACKGROUND Technical Field

This disclosure relates generally to techniques to detect anomalous ormalicious user behavior in an enterprise network.

Background of the Related Art

Enterprise security is a complex problem requiring the coordinationacross security policies, controls, threat models and threat detectionscenarios (use cases). The implementation of these policies, models andcontrols requires extensive use of threat monitoring technologies andsecurity devices, as well as human resources that have security,business and technical skills. In particular, the ever increasing numberof threats at scale requires automation in support of security analysts,who are responsible for preventing, detecting and responding to thesethreats. In most cases, the analyst must manually search through a widerange of data sources (some private, many public), review past threatevents and how they were handled, check for duplicate events, currentlyopen similar events and a knowledge database, etc., to determine anappropriate response procedure to handle this information. This processof data collection, analysis, and determining the final disposition ofthe alert, is time consuming and tedious for an analyst.

There are a variety of tools that exist for threat monitoring to analyzea wide range of data sources to identify patterns that are indicative ofthreats, security policy and control anomalies. When these threatsand/or anomalies are detected, actionable alerts are created. One suchtool is IBM® QRadar® User Behavior Analytics (UBA), which analyzes useractivity to detect malicious insiders and determine if a user'scredentials have been compromised. One application of such a system isto detect user behavior deviation from a user group, e.g., as defined ina user registry. A user registry associates each user with multipleattributes, e.g., in a Lightweight Directory Access Protocol (LDAP)application, users have attributes such as job title, department, and soforth. In this approach, users that have the same attributes areconsidered to be in the same group. The user behavior analytics systemdetects if a user's online behavior deviates from the defined group. Forexample, a deviation occurs if a person in the marketing department actsmore like a person in an engineering department.

A UBA system of this type adds user context to network, log,vulnerability and threat data to more quickly and accurately detectattacks. Using this tool, security analysts can easily see risky users,view their anomalous activities and drill down into the underlying logand flow data that contributed to a user's risk score.

While these systems provide significant advantages, there remains a needto provide more fine-grained approaches to detect user behaviordeviation.

BRIEF SUMMARY

This disclosure provides a machine learning-based technique for userbehavior analysis that detects when users deviate from expectedbehavior. In this approach, a set of user groups are provided,preferably based on information provided from a user registry. A set oftraining data for each of the set of user groups is then obtained,preferably by collecting security events generated for a collection ofthe users over a given time period (e.g., a last thirty (30) days). Amachine learning system is then trained using the set of training datato produce a model that includes a set of clusters in a model of userbehavior, wherein a cluster is a learned user group that corresponds toa defined user group. Once the model is built, it is then used toidentify users of a user group that deviate from their expected behaviorfor the group. In particular, the system compares a current behavior ofa user against the model and flags the user if the user's behavior isoutside what is expected.

In one implementation, the user behavior analysis is carried out inassociation with a Security Information and Event Management (SIEM)system In operation, the system takes input from a data server that logsuser activities, performs data analytics using the machine learning, andthen outputs user behavior deviation to a user interface to be consumedfor example by an end user (e.g., a security administrator). In thisapproach, the machine learning is used to obtain statistics for normalgroup behaviors, and to alert deviations from normal group behaviors.

The foregoing has outlined some of the more pertinent features of thesubject matter. These features should be construed to be merelyillustrative. Many other beneficial results can be attained by applyingthe disclosed subject matter in a different manner or by modifying thesubject matter as will be described.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the subject matter and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1 depicts an exemplary block diagram of a distributed dataprocessing environment in which exemplary aspects of the illustrativeembodiments may be implemented;

FIG. 2 is an exemplary block diagram of a data processing system inwhich exemplary aspects of the illustrative embodiments may beimplemented;

FIG. 3 illustrates a security intelligence platform in which thetechniques of this disclosure may be practiced;

FIG. 4 depicts a Level 1 security threat monitoring operation in a datacenter operating environment according to known techniques;

FIG. 5 depicts a Security Information Event Management (SIEM) platformthat includes a User Behavior Analytics (UBA) function in which thetechnique of this disclosure may be implemented;

FIG. 6 depicts a basic operation of the user behavior analyticstechnique of this disclosure by which the system detects if a user'sbehavior deviates from a defined LDAP group; and

FIG. 7 depicts a preferred machine learning technique used in thisdisclosure.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

With reference now to the drawings and in particular with reference toFIGS. 1-2, exemplary diagrams of data processing environments areprovided in which illustrative embodiments of the disclosure may beimplemented. It should be appreciated that FIGS. 1-2 are only exemplaryand are not intended to assert or imply any limitation with regard tothe environments in which aspects or embodiments of the disclosedsubject matter may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

With reference now to the drawings, FIG. 1 depicts a pictorialrepresentation of an exemplary distributed data processing system inwhich aspects of the illustrative embodiments may be implemented.Distributed data processing system 100 may include a network ofcomputers in which aspects of the illustrative embodiments may beimplemented. The distributed data processing system 100 contains atleast one network 102, which is the medium used to provide communicationlinks between various devices and computers connected together withindistributed data processing system 100. The network 102 may includeconnections, such as wire, wireless communication links, or fiber opticcables.

In the depicted example, server 104 and server 106 are connected tonetwork 102 along with storage unit 108. In addition, clients 110, 112,and 114 are also connected to network 102. These clients 110, 112, and114 may be, for example, personal computers, network computers, or thelike. In the depicted example, server 104 provides data, such as bootfiles, operating system images, and applications to the clients 110,112, and 114. Clients 110, 112, and 114 are clients to server 104 in thedepicted example. Distributed data processing system 100 may includeadditional servers, clients, and other devices not shown.

In the depicted example, distributed data processing system 100 is theInternet with network 102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers, consisting of thousands of commercial, governmental,educational and other computer systems that route data and messages. Ofcourse, the distributed data processing system 100 may also beimplemented to include a number of different types of networks, such asfor example, an intranet, a local area network (LAN), a wide areanetwork (WAN), or the like. As stated above, FIG. 1 is intended as anexample, not as an architectural limitation for different embodiments ofthe disclosed subject matter, and therefore, the particular elementsshown in FIG. 1 should not be considered limiting with regard to theenvironments in which the illustrative embodiments of the presentinvention may be implemented.

With reference now to FIG. 2, a block diagram of an exemplary dataprocessing system is shown in which aspects of the illustrativeembodiments may be implemented. Data processing system 200 is an exampleof a computer, such as client 110 in FIG. 1, in which computer usablecode or instructions implementing the processes for illustrativeembodiments of the disclosure may be located.

With reference now to FIG. 2, a block diagram of a data processingsystem is shown in which illustrative embodiments may be implemented.Data processing system 200 is an example of a computer, such as server104 or client 110 in FIG. 1, in which computer-usable program code orinstructions implementing the processes may be located for theillustrative embodiments. In this illustrative example, data processingsystem 200 includes communications fabric 202, which providescommunications between processor unit 204, memory 206, persistentstorage 208, communications unit 210, input/output (I/O) unit 212, anddisplay 214.

Processor unit 204 serves to execute instructions for software that maybe loaded into memory 206. Processor unit 204 may be a set of one ormore processors or may be a multi-processor core, depending on theparticular implementation. Further, processor unit 204 may beimplemented using one or more heterogeneous processor systems in which amain processor is present with secondary processors on a single chip. Asanother illustrative example, processor unit 204 may be a symmetricmulti-processor (SMP) system containing multiple processors of the sametype.

Memory 206 and persistent storage 208 are examples of storage devices. Astorage device is any piece of hardware that is capable of storinginformation either on a temporary basis and/or a permanent basis. Memory206, in these examples, may be, for example, a random access memory orany other suitable volatile or non-volatile storage device. Persistentstorage 208 may take various forms depending on the particularimplementation.

For example, persistent storage 208 may contain one or more componentsor devices. For example, persistent storage 208 may be a hard drive, aflash memory, a rewritable optical disk, a rewritable magnetic tape, orsome combination of the above. The media used by persistent storage 208also may be removable. For example, a removable hard drive may be usedfor persistent storage 208.

Communications unit 210, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 210 is a network interface card. Communications unit210 may provide communications through the use of either or bothphysical and wireless communications links.

Input/output unit 212 allows for input and output of data with otherdevices that may be connected to data processing system 200. Forexample, input/output unit 212 may provide a connection for user inputthrough a keyboard and mouse. Further, input/output unit 212 may sendoutput to a printer. Display 214 provides a mechanism to displayinformation to a user.

Instructions for the operating system and applications or programs arelocated on persistent storage 208. These instructions may be loaded intomemory 206 for execution by processor unit 204. The processes of thedifferent embodiments may be performed by processor unit 204 usingcomputer implemented instructions, which may be located in a memory,such as memory 206. These instructions are referred to as program code,computer-usable program code, or computer-readable program code that maybe read and executed by a processor in processor unit 204. The programcode in the different embodiments may be embodied on different physicalor tangible computer-readable media, such as memory 206 or persistentstorage 208.

Program code 216 is located in a functional form on computer-readablemedia 218 that is selectively removable and may be loaded onto ortransferred to data processing system 200 for execution by processorunit 204. Program code 216 and computer-readable media 218 form computerprogram product 220 in these examples. In one example, computer-readablemedia 218 may be in a tangible form, such as, for example, an optical ormagnetic disc that is inserted or placed into a drive or other devicethat is part of persistent storage 208 for transfer onto a storagedevice, such as a hard drive that is part of persistent storage 208. Ina tangible form, computer-readable media 218 also may take the form of apersistent storage, such as a hard drive, a thumb drive, or a flashmemory that is connected to data processing system 200. The tangibleform of computer-readable media 218 is also referred to ascomputer-recordable storage media. In some instances,computer-recordable media 218 may not be removable.

Alternatively, program code 216 may be transferred to data processingsystem 200 from computer-readable media 218 through a communicationslink to communications unit 210 and/or through a connection toinput/output unit 212. The communications link and/or the connection maybe physical or wireless in the illustrative examples. Thecomputer-readable media also may take the form of non-tangible media,such as communications links or wireless transmissions containing theprogram code. The different components illustrated for data processingsystem 200 are not meant to provide architectural limitations to themanner in which different embodiments may be implemented. The differentillustrative embodiments may be implemented in a data processing systemincluding components in addition to or in place of those illustrated fordata processing system 200. Other components shown in FIG. 2 can bevaried from the illustrative examples shown. As one example, a storagedevice in data processing system 200 is any hardware apparatus that maystore data. Memory 206, persistent storage 208, and computer-readablemedia 218 are examples of storage devices in a tangible form.

In another example, a bus system may be used to implement communicationsfabric 202 and may be comprised of one or more buses, such as a systembus or an input/output bus.

Of course, the bus system may be implemented using any suitable type ofarchitecture that provides for a transfer of data between differentcomponents or devices attached to the bus system. Additionally, acommunications unit may include one or more devices used to transmit andreceive data, such as a modem or a network adapter. Further, a memorymay be, for example, memory 206 or a cache such as found in an interfaceand memory controller hub that may be present in communications fabric202.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJava™, Smalltalk, C++ or the like, and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer, or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Those of ordinary skill in the art will appreciate that the hardware inFIGS. 1-2 may vary depending on the implementation. Other internalhardware or peripheral devices, such as flash memory, equivalentnon-volatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIGS. 1-2. Also, theprocesses of the illustrative embodiments may be applied to amultiprocessor data processing system, other than the SMP systemmentioned previously, without departing from the spirit and scope of thedisclosed subject matter.

As will be seen, the techniques described herein may operate inconjunction within the standard client-server paradigm such asillustrated in FIG. 1 in which client machines communicate with anInternet-accessible Web-based portal executing on a set of one or moremachines. End users operate Internet-connectable devices (e.g., desktopcomputers, notebook computers, Internet-enabled mobile devices, or thelike) that are capable of accessing and interacting with the portal.Typically, each client or server machine is a data processing systemsuch as illustrated in FIG. 2 comprising hardware and software, andthese entities communicate with one another over a network, such as theInternet, an intranet, an extranet, a private network, or any othercommunications medium or link. A data processing system typicallyincludes one or more processors, an operating system, one or moreapplications, and one or more utilities. The applications on the dataprocessing system provide native support for Web services including,without limitation, support for HTTP, SOAP, XML, WSDL, UDDI, and WSFL,among others. Information regarding SOAP, WSDL, UDDI and WSFL isavailable from the World Wide Web Consortium (W3C), which is responsiblefor developing and maintaining these standards; further informationregarding HTTP and XML is available from Internet Engineering Task Force(IETF). Familiarity with these standards is presumed.

Security Intelligence Platform with Incident Forensics

A known type of security intelligence platform is illustrated in FIG. 3.Generally, the platform provides search-driven data exploration, sessionreconstruction, and forensics intelligence to assist security incidentinvestigations. In pertinent part, the platform 300 comprises a set ofpacket capture appliances 302, an incident forensics module appliance304, a distributed database 306, and a security intelligence console308. The packet capture and module appliances are configured as networkappliances, or they may be configured as virtual appliances. The packetcapture appliances 302 are operative to capture packets off the network(using known packet capture (pcap) application programming interfaces(APIs) or other known techniques), and to provide such data (e.g.,real-time log event and network flow) to the distributed database 306,where the data is stored and available for analysis by the forensicsmodule 304 and the security intelligence console 308. A packet captureappliance operates in a session-oriented manner, capturing all packetsin a flow, and indexing metadata and payloads to enable fastsearch-driven data exploration. The database 306 provides a forensicsrepository, which distributed and heterogeneous data sets comprising theinformation collected by the packet capture appliances. The console 308provides a web- or cloud-accessible user interface (UI) that exposes a“Forensics” dashboard tab to facilitate an incident investigationworkflow by an investigator. Using the dashboard, an investigatorselects a security incident. The incident forensics module 304 retrievesall the packets (including metadata, payloads, etc.) for a selectedsecurity incident and reconstructs the session for analysis.

A representative commercial product that implements an incidentinvestigation workflow of this type is IBM® Security QRadar® IncidentForensics V7.2.3 (or higher). Using this platform, an investigatorsearches across the distributed and heterogeneous data sets stored inthe database, and receives a unified search results list. The searchresults may be merged in a grid, and they can be visualized in a“digital impression” tool so that the user can explore relationshipsbetween identities.

In particular, a typical incident forensics investigation to extractrelevant data from network traffic and documents in the forensicrepository is now described. According to this approach, the platformenables a simple, high-level approach of searching and bookmarking manyrecords at first, and then enables the investigator to focus on thebookmarked records to identify a final set of records. In a typicalworkflow, an investigator determines which material is relevant. He orshe then uses that material to prove a hypothesis or “case” to developnew leads that can be followed up by using other methods in an existingcase. Typically, the investigator focuses his or her investigationthrough course-grained actions at first, and then proceeds to fine-tunethose findings into a relevant final result set. The bottom portion ofFIG. 3 illustrates this basic workflow. Visualization and analysis toolsin the platform may then be used to manually and automatically assessthe results for relevance. The relevant records can be printed,exported, or submitted processing.

As noted above, the platform console provides a user interface tofacilitate this workflow. Thus, for example, the platform provides asearch results page as a default page on an interface display tab.Investigators use the search results to search for and access documents.The investigator can use other tools to further the investigation. Oneof these tools is a digital impression tool. A digital impression is acompiled set of associations and relationships that identify an identitytrail. Digital impressions reconstruct network relationships to helpreveal the identity of an attacking entity, how it communicates, andwhat it communicates with. Known entities or persons that are found inthe network traffic and documents are automatically tagged. Theforensics incident module 304 is operative to correlate taggedidentifiers that interacted with each other to produce a digitalimpression. The collection relationships in a digital impression reportrepresent a continuously-collected electronic presence that isassociated with an attacker, or a network-related entity, or any digitalimpression metadata term. Using the tool, investigators can click anytagged digital impression identifier that is associated with a document.The resulting digital impression report is then listed in tabular formatand is organized by identifier type.

Generalizing, a digital impression reconstructs network relationships tohelp the investigator identify an attacking entity and other entitiesthat it communicates with. A security intelligence platform includes aforensics incident module that is operative to correlate taggedidentifiers that interacted with each other to produce a digitalimpression. The collection relationships in a digital impression reportrepresent a continuously-collected electronic presence that isassociated with an attacker, or a network-related entity, or any digitalimpression metadata term. Using the tool, investigators can click anytagged digital impression identifier that is associated with a document.The resulting digital impression report is then listed in tabular formatand is organized by identifier type.

Typically, an appliance for use in the above-described system isimplemented is implemented as a network-connected, non-display device.For example, appliances built purposely for performing traditionalmiddleware service oriented architecture (SOA) functions are prevalentacross certain computer environments. SOA middleware appliances maysimplify, help secure or accelerate XML and Web services deploymentswhile extending an existing SOA infrastructure across an enterprise. Theutilization of middleware-purposed hardware and a lightweight middlewarestack can address the performance burden experienced by conventionalsoftware solutions. In addition, the appliance form-factor provides asecure, consumable packaging for implementing middleware SOA functions.One particular advantage that these types of devices provide is tooffload processing from back-end systems. A network appliance of thistype typically is a rack-mounted device. The device includes physicalsecurity that enables the appliance to serve as a secure vault forsensitive information. Typically, the appliance is manufactured,pre-loaded with software, and then deployed within or in associationwith an enterprise or other network operating environment;alternatively, the box may be positioned locally and then provisionedwith standard or customized middleware virtual images that can besecurely deployed and managed, e.g., within a private or an on premisecloud computing environment. The appliance may include hardware andfirmware cryptographic support, possibly to encrypt data on hard disk.No users, including administrative users, can access any data onphysical disk. In particular, preferably the operating system (e.g.,Linux) locks down the root account and does not provide a command shell,and the user does not have file system access. Typically, the appliancedoes not include a display device, a CD or other optical drive, or anyUSB, Firewire or other ports to enable devices to be connected thereto.It is designed to be a sealed and secure environment with limitedaccessibility and then only be authenticated and authorized individuals.

An appliance of this type can facilitate Security Information EventManagement (SIEM). For example, IBM® Security QRadar® SIEM is anenterprise solution that includes packet data capture appliances thatmay be configured as appliances of this type. Such a device isoperative, for example, to capture real-time Layer 4 network flow datafrom which Layer 7 application payloads may then be analyzed, e.g.,using deep packet inspection and other technologies. It providessituational awareness and compliance support using a combination offlow-based network knowledge, security event correlation, andasset-based vulnerability assessment. In a basic QRadar SIEMinstallation, the system such as shown in FIG. 3 is configured tocollect event and flow data, and generate reports. As noted, a user(e.g., an SOC analyst) can investigate offenses to determine the rootcause of a network issue.

Generalizing, Security Information and Event Management (SIEM) toolsprovide a range of services for analyzing, managing, monitoring, andreporting on IT security events and vulnerabilities. Such servicestypically include collection of events regarding monitored accesses andunexpected occurrences across the data network, and analyzing them in acorrelative context to determine their contribution to profiledhigher-order security events. They may also include analysis of firewallconfigurations, network topology and connection visualization tools forviewing current and potential network traffic patterns, correlation ofasset vulnerabilities with network configuration and traffic to identifyactive attack paths and high-risk assets, and support of policycompliance monitoring of network traffic, topology and vulnerabilityexposures. Some SIEM tools have the ability to build up a topology ofmanaged network devices such as routers, firewalls, and switches basedon a transformational analysis of device configurations processedthrough a common network information model. The result is a locationalorganization which can be used for simulations of security threats,operational analyses of firewall filters, and other applications. Theprimary device criteria, however, are entirely network- andnetwork-configuration based. While there are a number of ways to launcha discovery capability for managed assets/systems, and while containmentin the user interface is semi-automatically managed (that is, anapproach through the user interface that allows for semi-automated,human-input-based placements with the topology, and its display andformatting, being data-driven based upon the discovery of both initialconfigurations and changes/deletions in the underlying network), nothingis provided in terms of placement analytics that produce fully-automatedplacement analyses and suggestions.

FIG. 4 depicts a Security Operation Center (SOC) that provides Level 1security threat monitoring using an analytics platform 400 such as IBMQRadar. The platform 400 receives alerts (at step (1)) from a variety oflog sources 402, such as firewalls, intrusion detection and preventionsystems, antivirus systems, web proxies, and other systems and networkdevices. At step (2), the alerts are stored in an alert database 404. Atstep (3), the alerts are provided to a threat monitoring console 406that is manned by a security analyst 408. As is well-known, a SOCtypically is manned by different levels of security analysts. A Level 1(L1) analyst 408 is responsible for monitoring reported security events,and for closing or escalating those events according to SOC rules,policies and procedures. The security analyst 408 typically interactswith a client 410, which is the enterprise entity having an applicationthat is being monitored for security threats. Although not shown,typically the SOC has one or more additional levels of securityanalysts, such Level 2 (L2) and Level 3 (L3) analysts. Typically, L2security analysts handle escalations from L1 analysts and perform otheradministration and management functions, such as monitoring theperformance of the L1 analysts to ensure that security events arehandled timely, mentoring, and the like. Level 3 analysts handle furtherescalations (from L2 analysts), and provide additional higher-leveladministration and management functions in the SOC. Of course, thenumber of levels and the various tasks associated with each level may bevaried and implementation-specific.

As depicted, the L1 analyst makes a finding regarding an alert,typically with a goal of making this finding within about 15-20 minutesafter receiving the alert. Typically, the finding closes the alert (step5(a)) as a false positive, or escalation the alert (step 5(b)) as apossible attack. The false positive finding is stored in the alertdatabase 404. The attack finding typically is reported to the client 410whose application is affected. Depending on the implementation (e.g.,the SOC policy, the client procedure, etc.), some remediation or otheraction (step 6(b)) is taken; alternatively, the client 410 may indicatethat indeed the alert is a false positive and thus should be closed(step 6(c)). The responsive action 412 may be carried out in anautomated manner (e.g., programmatically), manually, or by a combinationof automation and manual operations. The action may be carried out bySOC personnel, by the client, or by a combination of SOC personnel andthe client. As also depicted, information regarding the response to thealert is also provided to a ticketing system 414, and such informationmay then be reported back to the security analyst (step 7(c)). Thesecurity analyst may then update the alert database (at step 8(c)) withthe information about how the alert was handled (or otherwise closed).Thus, the alert and its associated handling information is stored in thealert database 404 and available as a data source going forward.

By way of additional background, the enterprise typically includes anidentity management system by which the Company can define and manageorganizational role and access entitlement to resources. Role assignmentincludes assigning a user to one or more business roles in theorganization. Organizational roles are used to group people according totheir function in the organization. Thus, for example, all Companyemployees are granted the employee role in the organization. Typically,a user may be assigned to one or more organizational roles in a Company,such as, for example, site manager, project manager, HR manager, and thelike, as well as ancillary employee roles such as university liaison orcommunity coordinator. Assignment of a user to an organizational roleenables role-based provisioning of access entitlements to managedresources. For example, services in an identity manager representdifferent types of managed resources, such as Oracle® databases,Windows® machines, and the like. An organizational role may be linked toservices by means of provisioning policies, entitling persons in theorganizational role to an account on the managed resource that is linkedto that service.

The enterprise typically includes a directory service that hosts eachdirectory in the organization. Each such directory typically storesinformation about a particular application and the user(s) or group(s)of users that have entitlements to use that application, or anapplication instance. A representative directory is accessible via theLightweight Directory Access Protocol (LDAP), which is a directoryservice protocol that runs on top of the TCP/IP stack. LDAP provides aclient-server based mechanism that can be used to connect to, search,and modify network-accessible directories. Using LDAP, a directoryclient can query the directory application and obtain user/group data.

Detection of User Behavior Deviation from Defined User Groups

With the above as background, the following describes a user behavioranalysis and modeling technique according to this disclosure. As will bedescribed, the technique herein preferably leverages unsupervisedmachine learning to build a classification model that is then used todetect user behavior anomalies. Preferably, behavioral anomalies areflagged, e.g., by issuing an alert in a user interface, although theanalytics system may also interact with (and drive) other mitigation andremediation processing (perhaps dependent on a security policy). In atypical implementation, and as depicted in FIG. 5, the technique hereinis implemented in a UBA application 500 that executes in or inassociation with at SIEM 502. The application may be implemented as aset of computer program instructions executed by one or more hardwareprocessors, e.g., in a data processing system such as depicted in FIG. 2and described above.

As will be described, the UBA application 500 implements a machinelearning (ML) system that detects user behavior deviation from one ormore user groups defined in a user registry 504. In particular, theapplication 500 takes input from a data server 506 that logs useractivities, performs data analytics, and output user behavior deviationalerts to a user interface 508, where they are consumed by an end user(e.g., a security analyst). As will be described, unsupervised machinelearning is used to obtain statistics for normal group behaviors, and toalert deviations from normal group behaviors.

To this end, FIG. 6 depicts the basic operating principle of thebehavior analysis and modeling technique of this disclosure. As depictedin the top half FIG. 6, and based on historical data, two distinct LDAPgroups 600 and 602 have been identified and are depicted as includingvarious members. By way of example, LDAP group 600 represent anengineering group, whereas LDAP group 602 represents a marketing group.More generally, and in this approach, users are grouped, preferably bytheir user registry attributes, such as job title, department and thelike. Typically, members in each defined LDAP group (e.g., groups 600and 602) are expected to behave in a relatively similar manner withrespect to their expected contribution to log activity; conversely,members of different LDAP groups generally behave differently from oneanother. Thus, and continuing with this example, a relatively largerpercentage of members in the engineering group 600 would be expected toaccess a source code control system or a coding discussion forum ascompared to the percentage of users in the marketing group that mightundertake such activities. Conversely, a relatively larger percentage ofusers in the marketing group would be expected to access a salesplatform application as compared to the percentage of users from theengineering group that might do so. As will be seen, the technique ofthis disclosure exploits these differences in access patterns tofacilitate a determination regarding user behavior being analyzed. Thus,and as depicted in the bottom portion of FIG. 6, the basic operation ofthe system detects if the behavior of a user 604 deviates from his orher defined LDAP group. To this end, the system clusters users,preferably using real access pattern data, and then flags the user 604(from the engineering LDAP group 600) for deviating towards themarketing LDAP group 602. If such deviation is identified, a givenaction (e.g., issuing an alert) is taken.

There is no limitation on the number of groupings, or their constituentsor attributes or other properties. Typically, and as noted above,groupings are based on LDAP attributes, although this is not alimitation either, as any grouping mechanism that identifies grouping(s)based on expected user contributions to log activity may be used.

FIG. 7 depicts a preferred machine learning (ML) system that is used tosupport the behavior analytics approach of this disclosure. As will beseen, machine learning preferably occurs in a two (2)-phase manner thatis now described in detail.

At step (1), and as described above, users are grouped, preferably usingtheir LDAP properties, into LDAP groupings 700. For example, and usingjob title and department as example attributes, users in a marketingdepartment with an analyst title are considered to be in the same LDAPgroup 700. Generalizing, group information is represented using a vectorof set G=(S₁,S₂, . . . , S_(n-1),S_(n))

${\sum\limits_{i = 1}^{n}\; {S_{i}}} = N$

where a is the number of LDAP groups, S_(i) is a set of users in LDAPgroup i, and is a total number of users.

With the LDAP grouping 700 in hand, a training set is then obtained. Tothis end, and at step (2), preferably low-level category security eventsare ingested from a log server 702 for these users, e.g., for a pastperiod of T days. A representative value of the variable T is thirty(30) (for the past 30 days), but this is not a limitation, as shorter orlonger periods may be used. For each user and each day, there is a setof slice data 704. In this step, user is activities U_(i) measured onthe wide range of low-level categories are then represented as follows:

U _(i)=(C ₁ ^(i) ,C ₂ ^(i) , . . . ,C _(L-1) ^(i) ,C _(L) ^(i)), i=1,2,. . . ,N

C _(j) ^(i)=(C _(j,1) ^(i) ,C _(j,2) ^(i) , . . . ,C _(j,t-1) ^(t) ,C_(j,r) ^(i)),i=1,2, . . . ,N;j=1,2, . . . ,L

where, C_(j) ^(i) denotes user i's log activity in category j, and L isthe total number of low-level categories. In practice, L typically is arelatively large number, e.g., L=2000.

The nature and type of low-level categories will vary depending onimplementation. By way of background, a product such as IBM QRadargroups log sources into high-level categories, with each high-levelcategories comprising a set of low-level categories. Each event isassigned a specific high-level category and a low-level category withinthat high-level category. A representative high-level category may be“Application,” which represents events that are related to applicationactivity, and that high-level category typically includes a large numberof low-level categories corresponding to log events that may occur withrespect to a user's interaction with one or more applications thatcorrespond to the Application type. In the case of an email application,for example, the low-level categories may include Mail Opened, MailClosed, Mail Terminated, Mail Denied, Mailed Queued, and so forth. Eachlow-level category has as description and severity level that iscaptured in the log. The slice data 704 thus represents a data set oflow-level security events derived for each user in the grouping.

Thereafter, and at step (3), a first training phase is initiated. Inthis phase, a model is built. Preferably, the model is a LatentDirichlet Allocation (LDA) model 708 that is built on an entire data set{U_(i)}_(i=1, 2 . . . , N) to infer the activity patterns from all thelog events. LDA modeling 706 is a generative statistical modelingtechnique that allows sets of observations to be explained by unobservedgroups that explain why some parts of the data are similar. In LDA, each“document” may be viewed as a mixture of various topics where eachdocument is considered to have a set of topics that are assigned to itvia LDA. In the context of the machine learning technique of thisdisclosure, a “document” represents the low-level log data collected fora particular user of a particular day (the slice data 704 as previouslymentioned). Thus, if there are 10 users and 30 days of interest, thereare 300 documents.

In LDA, a number of patterns represented in the model is a modelparameter, and preferably it is learned through model selection viaAkaike Information Criterion (AIC). AIC is an estimator of the relativequality of statistical models for a given set of data. In particular,and given a collection of models for the data, AIC estimates the qualityof each model, relative to each of the other models. Thus, AIC providesa means for model selection.

While LDA with AIC model selection is a preferred approach, this is nota limitation. Alternatives to LDA include models that are built, forexample, using latent semantic indexing, independent component analysis,probabilistic latent semantic indexing, non-negative matrixfactorization, and Gamma-Poisson distribution. An alternative to AICthat may be used is Bayesian Information Criteria (BIC).

According to a preferred approach, the number of patterns preferably isselected from a model that produces a minimum AIC. The result is the LDAmodel 708 with K patterns, wherein K generally falls in the range of 2to 20. The process then continues at step (4). At this step, thetraining data (namely, the users' activities) is transformed, preferablyby applying the learned LDA model, to produce the following:

U _(i)=(D ₁ ^(i) ,D ₂ ^(i) , . . . ,D _(K-1) ^(i) ,D _(K) ^(i)), i=1,2,. . . ,N

D _(j) ^(i)=(C _(j,1) ^(i) ,C _(j,2) ^(i) , . . . ,C _(j,T-1) ^(i) ,C_(j,T) ^(i)), i=1,2, . . . ,N,j=1,2, . . . ,K

This operation effectively reduces the dimension of features in the datafrom L to K. Typically, K is much smaller than L, and typically K rangesfrom 2 to 20, although this is not a limitation.

The process then continues with training moving to a second phase. Inparticular, and at step (5), and using the transformed training data,and for each day in {1, 2, . . . , T}, the users are clustered into nclusters (typically, the total number of LDAP groups) using a clusteringalgorithm, for example, a Gaussian Mixture Model. In statistics, amixture model is a probabilistic model for representing the presence ofsub-populations within an overall population, without requiring that anobserved data set should identify the sub-population to which anindividual observation belongs. Formally, a mixture model corresponds tothe mixture distribution that represents the probability distribution ofobservations in the overall population. The clustering 710 produces aninferred grouping structure as a vector of set

G′=(S ₁ ′,S ₂ ′, . . . ,S _(n-1) ′,S _(n)′).

Alternatives to GMM include K-means, X-means, CobWeb, Density-BasedSpatial Clustering of Applications with Noise (DBSCAN).

The routine then continues at step (6). At this step, and for each useri∈{1, 2, . . . , N} in each day, the process finds a user's inferred(learned) group in G′, (S′∈G′), and the user's LDAP group in G, (S∈G),and then computes a measure of the group difference, preferably using adistance measure (e.g., Jaccard distance 712) computed as follows:

${D_{I}\left( {S,S^{\prime}} \right)} = \frac{{{S\bigcup S^{\prime}}} - {{S\bigcap S^{\prime}}}}{{S\bigcup S^{\prime}}}$

Other distances measures may be used as well. To provide a concreteexample, for a user (user 1), the user's learned group may comprise user1, user 2, user 3 and user 4, whereas the user's LDAP group, e.g.,comprises user 1, user 2, user 4 and user 10.

The process then continues at step (7). At this step, an empiricaldistribution 714 for the Jaccard distances (D_(J)) is computed acrossall users and all dates. To this end, a cumulative distribution function(CDF) of the empirical distribution is denoted as:

F _(D) _(j) (d)=P(D _(J) ≤d),

which is the probability that D_(I) is less than or equal to d. The CDFtypically is a large distribution, and it comprises a number ofclusters, with each cluster typically comprising a set of useridentifiers (e.g., usernames). Clustering preferably is carried out foreach user and for each day represented in the modeling period. Eachcluster in the CDF represents a learned user group and is based on thenetwork event statistics that are accumulated over the relevant timeperiod (e.g., the last 30 days). This completes the second phase of themachine learning training, and thus the training phase as a whole.

In a representative embodiment, the above-described machine learning iscarried out periodically, e.g., every seven (7) days, with the resultingCDF then being used for the next week to alert deviations from normalgroup behaviors. As noted, preferably thirty (30) days of log event datais used to build and refine the machine learning model. These timeperiods of course are merely exemplary, as the model (and the resultingCDF) re-computed more or less frequently, and using data sets other than30 days. Irrespective of the frequency that the ML model is built andthe CDF computed, the CDF is then used with respect to new data (e.g.,for a given day, or other time period in question) to determine whethera particular user's behavior should be flagged for a deviation.

In particular, and at step (9), and when a new day's data comes in(assuming real-time data is received and analyzed by the system on adaily basis), the new data slice is then scored at 716. To this end,preferably the above-described processing in steps (4)-(6) is repeatedto apply the LDA model to transform the data, cluster the users, and forany user, calculate the Jaccard distance between the user's inferred(learned) cluster to the user's actual LDAP group, and denote thedistance as d. The user is then flagged (as a user who has deviated fromhis or her user group) if F_(D) _(J) (d)≥P_(t), where P_(t) is athreshold, e.g., P_(t)=0.99. This percentage value of course is merelyrepresentative and is not intended to limit the technique. In such case,preferably a given action is taken, e.g., an alert is delivered to auser interface. Other response actions include sending a notification,performing a mitigation operation, restricting the user from accessing aresource, or the like. This competes the processing.

Thus, according to this disclosure, users are identified with one ormore defined groups (e.g., based on information in the directory) andalso potentially with groups that are learned by the machine learning.The defined groups are fixed and determined, for example, by LDAP jobtitle (or some other criteria). Thus, assume there are 10 users, withusers [1-5] being engineers, and users [6-10] being assigned tosales-related positions. In the defined group, users 1-5 are in an LDAPgroup, and users 6-10 are in another LDAP group. As noted above, membersof an LDAP group are anticipated to share the same online behavior. Whenthe machine learning evaluates the real online behavior of theseindividuals, assume user 5 behaves more like a salesperson (than anengineer); this leads to two learned groups being developed by themachine learning, namely, one group that contains users [1-4], and theother group that contains users [5, and 6-10]. Jaccard distances betweenthe defined group and the learned group are used to calculate thecumulative distributed function (CDF) that is then used to facilitatethe discrimination and alerting. As noted above, preferably the Jaccarddistance history that is used to estimate the CDF is calculated acrossall users and all dates. In this approach, if there are only a fewoutliers, the members in the learned group and an LDAP group shouldalmost be the same (but with some difference due to the few outliers).For each user, the approach as described calculates his or her deviationfrom his or her LDAP group. For an arbitrary user, his or her deviationfrom his or her LDAP group is measured through the Jaccard distancebetween the membership (the users) in his or her learned group and themembership in his or her LDAP group. The distance is further evaluatedon the CDF plotted over preferably all the historical data. Only whenthe Jaccard distance is large enough to render it larger than a majorityof all historical Jaccard distances (using a configurable threshold,e.g., 99%), an alert for this user is triggered. The high percentile isdesirable to reduce false alarms.

In operation, if a particular user is behaving on a given day (or someother time period being evaluated), his or her learned group and LDAPgroup should coincide (and there should be no alert generated). If,however, the particular user's behavior is sufficiently deviant, his orher learned group and LDAP group typically will differ, and depending onthe degree of that difference (given the configurable threshold), analert is generated.

In this approach, if there are only a few outliers, the members in thelearned group and an LDAP group should almost be the same (but with somedifference due to the few outliers). For each user, the approach asdescribed calculates his or her deviation

The approach as described above provides significant advantages. Thetechnique enables robust detection of user behavior abnormalities and,in particular, based on the notion of detecting deviation from definedgroups. The approach is fine-grained as it groups the users according toa preferably two-stage machine learning system. Preferably, registryinformation from a directory system is used to provide the number ofclusters and the basis for comparison. The system learns by carrying outclustering on data slices derived from network event data. The techniqueis fine-grained as it provides for user grouping preferably using latentallocation and a Gaussian mixture model. The technique is readilyimplemented, e.g., as an application in a SIEM platform, although it mayalso be provided as a standalone application. As has been described, theapproach herein leverages data ingested from the SIEM for low-levelsecurity event categories to provide a baseline (defined) clustering,preferably using users' directory (e.g., LDAP)) properties to facilitatethe machine learning system, and then computes statistical deviationsfrom the defined grouping(s) with preferably real-time data; behaviorabnormalities are then output, e.g., in a graphic user interface (GUI),to an interested user. As one of ordinary skill will recognize, thetechnique leverages an unsupervised learning system and thus does notneed labeled data.

More generally, the technique herein provides for an enhanced userbehavior analytics system that can detect insider threats, helpingsecurity analysts detect anomalous or malicious behaviors that occur onthe network.

This subject matter may be implemented as-a-service. The subject mattermay be implemented within or in association with a data center thatprovides cloud-based computing, data storage or related services. Themachine learning (ML) functionality may be provided as a standalonefunction, or it may leverage functionality from other ML-based productsand services.

In a typical use case, a SIEM or other security system has associatedtherewith a user interface that can be used to render the alertvisually, to search and retrieve relevant information from alertdatabase, and to perform other known input and output functions withrespect thereto.

As noted above, the approach herein is designed to be implemented in anautomated manner within or in association with a security system, suchas a SIEM.

The functionality described in this disclosure may be implemented inwhole or in part as a standalone approach, e.g., a software-basedfunction executed by a hardware processor, or it may be available as amanaged service (including as a web service via a SOAP/XML interface).The particular hardware and software implementation details describedherein are merely for illustrative purposes are not meant to limit thescope of the described subject matter.

More generally, computing devices within the context of the disclosedsubject matter are each a data processing system (such as shown in FIG.2) comprising hardware and software, and these entities communicate withone another over a network, such as the Internet, an intranet, anextranet, a private network, or any other communications medium or link.The applications on the data processing system provide native supportfor Web and other known services and protocols including, withoutlimitation, support for HTTP, FTP, SMTP, SOAP, XML, WSDL, UDDI, andWSFL, among others. Information regarding SOAP, WSDL, UDDI and WSFL isavailable from the World Wide Web Consortium (W3C), which is responsiblefor developing and maintaining these standards; further informationregarding HTTP, FTP, SMTP and XML is available from Internet EngineeringTask Force (IETF). Familiarity with these known standards and protocolsis presumed.

The scheme described herein may be implemented in or in conjunction withvarious server-side architectures including simple n-tier architectures,web portals, federated systems, and the like. The techniques herein maybe practiced in a loosely-coupled server (including a “cloud”-based)environment.

Still more generally, the subject matter described herein can take theform of an entirely hardware embodiment, an entirely software embodimentor an embodiment containing both hardware and software elements. In apreferred embodiment, the function is implemented in software, whichincludes but is not limited to firmware, resident software, microcode,and the like. Furthermore, as noted above, the identity context-basedaccess control functionality can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer readable medium can be any apparatus thatcan contain or store the program for use by or in connection with theinstruction execution system, apparatus, or device. The medium can be anelectronic, magnetic, optical, electromagnetic, infrared, or asemiconductor system (or apparatus or device). Examples of acomputer-readable medium include a semiconductor or solid state memory,magnetic tape, a removable computer diskette, a random access memory(RAM), a read-only memory (ROM), a rigid magnetic disk and an opticaldisk. Current examples of optical disks include compact disk-read onlymemory (CD-ROM), compact disk-read/write (CD-R/W) and DVD. Thecomputer-readable medium is a tangible item.

The computer program product may be a product having programinstructions (or program code) to implement one or more of the describedfunctions. Those instructions or code may be stored in a computerreadable storage medium in a data processing system after beingdownloaded over a network from a remote data processing system. Or,those instructions or code may be stored in a computer readable storagemedium in a server data processing system and adapted to be downloadedover a network to a remote data processing system for use in a computerreadable storage medium within the remote system.

In a representative embodiment, the threat disposition and modelingtechniques are implemented in a special purpose computer, preferably insoftware executed by one or more processors. The software is maintainedin one or more data stores or memories associated with the one or moreprocessors, and the software may be implemented as one or more computerprograms. Collectively, this special-purpose hardware and softwarecomprises the functionality described above.

While the above describes a particular order of operations performed bycertain embodiments of the invention, it should be understood that suchorder is exemplary, as alternative embodiments may perform theoperations in a different order, combine certain operations, overlapcertain operations, or the like. References in the specification to agiven embodiment indicate that the embodiment described may include aparticular feature, structure, or characteristic, but every embodimentmay not necessarily include the particular feature, structure, orcharacteristic.

Finally, while given components of the system have been describedseparately, one of ordinary skill will appreciate that some of thefunctions may be combined or shared in given instructions, programsequences, code portions, and the like.

The techniques herein provide for improvements to another technology ortechnical field, e.g., security incident and event management (SIEM)systems, other security systems, as well as improvements toautomation-based cybersecurity analytics.

Having described the invention, what we claim is as follows:
 1. A methodfor detecting user behavior deviation in an enterprise network,comprising: defining a set of user groups; receiving security events fora collection of users from an application and generating a set oftraining data; training a machine learning system using the set oftraining data set to generate a set of clusters in a model of userbehavior, wherein a cluster is a learned user group that corresponds toa defined user group in the set of user groups; and comparing a behaviorof a user against the model to detect a deviation from an expectedbehavior for the user as represented by one of the clusters.
 2. Themethod as described in claim 1 wherein the set of user groups aredefined based on user attribute information received from a userdirectory.
 3. The method as described in claim 1 wherein the applicationis a security information and event monitoring (SIEM) application. 4.The method as described in claim 3 wherein the security events compriselow-level security events ingested from the SIEM.
 5. The method asdescribed in claim 1 wherein the set of clusters is generated by:transforming the set of training data according to a latent allocationmodel to generate transformed training data; applying a Gaussian mixturemodel to the transformed training data to derive a set of learnedgroupings; and computing measures of distance between the set of learnedgroupings and the set of user groups to generate a probabilitydistribution function.
 6. The method as described in claim 5 whereincomparing the behavior of the user includes: receiving new datarepresenting security events associated with a set of users that includethe user; re-applying the latent allocation model and the Gaussianmixture model to the new data; for the user, computing a measure ofdistance between a learned grouping that includes the user and the usergroup associated with the user; and determining whether the measure ofdistance computed for the user exceeds a given probability threshold. 7.The method as described in claim 6 wherein the measure of distancecomputed for the new user is a Jaccard distance.
 8. An apparatus,comprising: a processor; computer memory holding computer programinstructions executed by the processor to detect user behavior deviationin an enterprise network, the computer program instructions configuredto: define a set of user groups; receive security events for acollection of users from an application and generating a set of trainingdata; train a machine learning system using the set of training data setto generate a set of clusters in a model of user behavior, wherein acluster is a learned user group that corresponds to a defined user groupin the set of user groups; and compare a behavior of a user against themodel to detect a deviation from an expected behavior for the user asrepresented by one of the clusters.
 9. The apparatus as described inclaim 8 wherein the set of user groups are defined based on userattribute information received from a user directory.
 10. The apparatusas described in claim 8 wherein the application is a securityinformation and event monitoring (SIEM) application.
 11. The apparatusas described in claim 10 wherein the security events comprise low-levelsecurity events ingested from the SIEM.
 12. The apparatus as describedin claim 8 wherein the computer program instructions to generate the setof clusters includes computer program instructions further configuredto: transform the set of training data according to a latent allocationmodel to generate transformed training data; apply a Gaussian mixturemodel to the transformed training data to derive a set of learnedgroupings; and compute measures of distance between the set of learnedgroupings and the set of user groups to generate a probabilitydistribution function.
 13. The apparatus as described in claim 12wherein the computer program instructions to compare the behavior of theuser includes computer program instructions further configured to:receive new data representing security events associated with a set ofusers that include the user; re-apply the latent allocation model andthe Gaussian mixture model to the new data; for the user, compute ameasure of distance between a learned grouping that includes the userand the user group associated with the user; and determine whether themeasure of distance computed for the user exceeds a given probabilitythreshold.
 14. The apparatus as described in claim 13 wherein themeasure of distance computed for the new user is a Jaccard distance. 15.A computer program product in a non-transitory computer readable mediumfor use in a data processing system to detect user behavior deviation inan enterprise network, the computer program product holding computerprogram instructions that, when executed by the data processing system,are configured to: define a set of user groups; receive security eventsfor a collection of users from an application and generating a set oftraining data; train a machine learning system using the set of trainingdata set to generate a set of clusters in a model of user behavior,wherein a cluster is a learned user group that corresponds to a defineduser group in the set of user groups; and compare a behavior of a useragainst the model to detect a deviation from an expected behavior forthe user as represented by one of the clusters.
 16. The computer programproduct as described in claim 15 wherein the set of user groups aredefined based on user attribute information received from a userdirectory.
 17. The computer program product as described in claim 15wherein the application is a security information and event monitoring(SIEM) application.
 18. The computer program product as described inclaim 17 wherein the security events comprise low-level security eventsingested from the SIEM.
 19. The computer program product as described inclaim 15 wherein the computer program instructions to generate the setof clusters includes computer program instructions further configuredto: transform the set of training data according to a latent allocationmodel to generate transformed training data; apply a Gaussian mixturemodel to the transformed training data to derive a set of learnedgroupings; and compute measures of distance between the set of learnedgroupings and the set of user groups to generate a probabilitydistribution function.
 20. The computer program product as described inclaim 19 wherein the computer program instructions to compare thebehavior of the user includes computer program instructions furtherconfigured to: receive new data representing security events associatedwith a set of users that include the user; re-apply the latentallocation model and the Gaussian mixture model to the new data; for theuser, compute a measure of distance between a learned grouping thatincludes the user and the user group associated with the user; anddetermine whether the measure of distance computed for the user exceedsa given probability threshold.
 21. The computer program product asdescribed in claim 20 wherein the measure of distance computed for thenew user is a Jaccard distance.